Papers with human-LLM collaboration pipeline
Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principles (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for improving LLMs in simulations are limited due to privacy concerns and limited domain knowledge. |
| Approach: | They propose a pipeline that elicits qualitative feedback from a domain-expert and transforms it into a set of principles that govern an LLM-prompted roleplay. |
| Outcome: | The proposed pipeline shows a 30% improvement in response quality and principle following for the downstream task. |
DisastQA: A Comprehensive Benchmark for Evaluating Question Answering in Disaster Management (2026.findings-acl)
Copied to clipboard
Zhitong Chen, Kai Yin, Xiangjue Dong, Chengkai Liu, Xiangpeng Li, Bo Li, Junwei Ma, Yiming Xiao, Ali Mostafavi, James Caverlee
| Challenge: | Existing benchmarks for question answering (QA) are lacking in a high-stakes environment. |
| Approach: | They propose a rigorously verified benchmark of 3,000 expert-annotated questions . they propose 'keypoint-based evaluation protocol' emphasizing factual completeness over verbosity . |
| Outcome: | Experiments with 20 models reveal substantial divergences from general-purpose models such as MMLU-Pro. |